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ABSTRACT 

The effects of correlated dimensions on parameter 
estimation were assessed, using a two-dimensional item response 
theory model. Past research has shown the inadequacies of the 
unidimensional analysis of multidimensional item response data. 
However, few studies have reported multidimensional analysis of 
multidimensional data, and, in those using simulated data, the 
results were usually based on replication. Multidimensional analysis 
of simulated two-dimensional item response data fitting the M2PL 
model of M. D. Reckase (1985) was done using the analysis program 
known as MIRTE. A Monte Carlo study was employed. Three data sets 
(2,000 ability vectors by 104 items) were generated to satisfy 
different degrees of correlation between the two abilities. The ^ ata 
sets and analyses were replicated 100 times each. Summary statis^ Ics 
on the 100 replications were used to examine the effects of the 
degree of correlation between ability dimensions. Results shed li 
on the degree of correlation between the two ability dimensions. 
Ability and item parameters were recovered well enough to encourage 
further investigation of and to justify limited use of 
multidimensional analysis . (Author/TJH) 
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Abstract 

The purpose of this study was to assess the effects of correlated dimensions on 
parameter estimation using a cvo-dimensionai IRT moael, r^asi researcn has snovn 
the inadequacies of unldimensionai analysis of multidimensional item response data. 
However, few studies have reported multidimensional analysis of multidimensional 
data and, in those which used simulated data, results were usually based on one 
replication. 

Multidimensional analysis of simulated two-dimensional item response data fitting 
the M2PL model of Reckase (igsSa) was done using the analysis program, MIRTE 
(Carlson, 1987). 

Three data sets (2000 ability vectors by 104 items) were generated to satisfy 
differ'^nt degress of correlation bet^reen the tT^ro abilities. The three data sets and 
analyses were replicated 100 times each. Summary statistics on the 100 replications 
were used to assess the effects of degree of correlation between ability dimensions. 

Results indicated that the degree of correlation between the two ability dimensions 
was less well recovered as p(6i,e2) increased. Ability and item parameters were 

recovered well enough to encourage further investigation of and to justify limited use 
of multidimensional analysis. 
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Theoretical Framework 

The ongmal Item Response Thaory ^IRT) models were based on :he assumption or 
unidimensionallty (i.e., only one abiUty was required to correctly respond to all the 
items). When more than one ability accounts for test performance, the test is 
multiaimensional and a Multidimensional Item Response Theory (MIRT) model is 
required to accurately fit the data. 

Consider the situation In which itens fojr a test are designed to measure one ability 
(e.g., mathematics) but require some amount of a second ability (e.g., verbal) in order 
to respond correctly. This seccna, required ability could be more crucial to success for 
some examinees than others. For example, students of English as a Second Language 
(ESL) may have sufficient mathematics ability but lack the required amount of verbal 
ability in order to make a correct response. It is reasonable to assume the two abilities 
are correlated to some extent. What happens to ability estimates if a MIRT model Is 
used to fit the responses? How are the ability estimates affected by degree of 
correlation between the abilities? 

Several authors (e,g,, Ansley & Forsyth, 1985; Began & Yen; i983"^ Dorans & 
Kingston, 1985; Drasgow & Parsons, 1983; McCauley & Mendosa, 1985; McKinley & 
Reckase, 1984; Eeckase> 1979, 1985b; Reckase> Carlson, Ackerrnan & Spray, 1986) have 
considered the effects of analyzing known multidimensional data with a 
unidimensional item response model. The resulting estimates in most cases were not 
acceptable unless there was clearly one dominant dimension, Ansley and Forsyth 
(1985) reported that the unidimensional ability estimates were most highly related to 
the average of the multidimensional abilities. In the hypothiitical educational situation 
described above, this would be unacceptable if students with high mathematics abihty 
but low verbal ability were penalized in placement or selection procedures, Reckase et 
al(1986) found that the unidimensional ability estimates established from 
multidimensional data had different interpretations cit different points on the 
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Unidlmensional ability scale. By and large, the resulting uni dimensional estimates 
from multidimensional data have been difficult to interpret and have not reflected -veil 
the original characteristics of the data. 

In spite of findings that unldlmensional models are not often robust to 
multldimensionality, few researchers have made use of multidimensional models to 
analyze multidimensional data. There are good reasons for this. Although MIRT 
models are being developed and tested, they are more complex than their 
unldlmensional counterparts. Analysis of multidimensional data with 
multidimensional programs is expensive in terms of computer time. Few 
multidimensional analysis programs exist and none has undergone exhaustive testing. 
Only two programs have been readily available; (l) TESTFACT (Wilson, Wood & 
Gibbons, 1984); and (2) MAXLOG (McKinley & Reckase, 1983b). TESTFACT has been 
deemed inappropriate by some researchers because it uses a linear factor analytic 
procedure to describe the non-linear IRT relationship, a particularly contentious 
procedure with multidimensional data (Ansley, 1984; Lord, 1980; McDonald & Ahlawat, 
1974; R. L McKinley, personal communication, November 13, 1986), MAXLOG was 
written to provide parameter estimates for uncorrelated abilities. Results of pilot 
testing of a third multidimensional analysis program, MIRTE (Carlson, 1987), indicate 
that it estimates item parameters and abilities more efficiently and more accurately 
than MAXLOG and it can accommodate data from correlated dimension^. The program 
is designed to analyze data which fit the multidimensional two-parameter logistic ogive 
(M2PL) model (McKinley & Reckase, 1983ai Reckase, 1985b, 1986). 

It is unreasonable to assume abilities are uncorrelated for most achievement tests. 
McKinley and Reckase (1984) considered the effects of analyzing data generated for 
correlated dimensions using MAXLOG. The ability and item estimates were confounded 
in the results of the data analysis. However, when the underlying abilities were 



correlated and a unidimensional analysis was used, again both uni dimensional ability 
and Item parameter e*5timates vere aifacted (McKiniey & Reckase. :984). 

Researchers who have used multidimensional analysis (e*g., McKiniey, 1983; 
McKinley & Reckase, 1983a, 1983b, 1984; Muraki & Englehard, 1985) have indicated 
that a multidimensional model more adequately describes both real and simulated 
muttldimensloQal data than does a unidimensional model. However, in most cases, the 
simulation studies have been based on no replications so that stability of estimates is 
difficult to determine. There is a need to know how consistently these estimates are 
recovered. The effects of both correlated abilities and differential secondary ability on 
parameter estimation need to be evaluated in a comprehensive, systematic manner. 

Purpose of the Study 

The purpose of this study was lo determine the adequacy of multidimensional 
ability and item parameter estimates using a MIRT analysis. Specifically the question 
to be addressed is: w^n^t is the effect of correlated abibty dimensions on parameter 
estimation for a two-parameter, two-dlmensionai IRT n;odel? 

Methodology 

A Monte Carlo study was chosen to answer the research questions. 

Model Description 

The data for the study were generated to fit the multidimensional two-parameter 
logistic (M2PL) model (McKinley $c Reckase, 1983a) which was updated by Reckase 
(1985b, 1986), A description of the updated'version foUows^ 

The mathematical formula is given by Equation (I). 

exp (ai'Sj + di) 

Pjj = P(xij - 1 Up dj, Sj) = , (1) 

1 + exp (aj^Sj + di) 
(i = 1, 2, . _ , n, j =^ 1, 2, . . N) 



•vhere ?ij is the probability of a correct response to item * '^y examinee ji x,. is the 
response (1 = correct; 0 - incorrect) of examinee j on item ii is a vector of m 
discrimination parameters; di is a parameter representing the difficulty ot item i; is 
a vector of m ability parameters for individual j; N is the number of examinees; n :s 
the number of items; and m is the number of dimensions. 

This irodel is compensatory in that it allows high proficiency on one dimension to 
compensate for lo-w proficiency on other dimensions in arriving at a correct response to 
a test item, 

Reckase (1986) defined a multidimensional discrimination parameter for item i to be 

m 0.5 
MDlSCi = [ 2 (aji^)^ ] , (2) 
k=l 

This parameter is related to the item characteristic curve on the multidimensional iten: 
response surface above the line through the origin of the ability space and to the point 
oi maximum information and is therefore analogous to the unidimensional 
discrimination parameter (Carlson, 1987), 

Reckase (1985b) also defined a multidimensional item difficulty parameter* MDIFj, 

such that 

m 0.5 

MDIFj = -di / [2 (aik)2] . (3) 
k=l 

= -dj / MDlSCi 

This parameter represents the distance between the origin of the m-dimensiona! ability 
space and the point in the space where the item information is a maximum. The line 
joining this point to the origin is at an angle of aji^ to the k^ ability dimension where 



m 0,5 

cos ocjjt = a^jt 

Program Der>crlPtlon 

The program used to analyse the two-dimensional data was MIRTE (Carlson, 1S87)- 
While written specifically to provide estimates of item and ability parameters for a 
M3PL model, the program Is readily adapted to the M2PL model by letting the third 
item parameter equal zero. As well as estimation of abilities, item discriminations and 
difficulty, MIRTE provides estimates of standard errors for each of the parameter 
estimates. Estimates of the multidimensional item difficulty and discrimination are 
also provided. The method of estimation used is a variation of the joint maximum 
likelihood procedure using a modified Newton-Raphson iieration technique and the 
algorithm used is similar to that used In the unidlmensional analysis program, LOGIST 
(Wingersky, Barton, & Lord, 1982). The MIRTE (version 2.00) used in this study was 
found to estimate parameters when dimensions were correlated better than MAXLOG 
(J. E. Carlson, personal communication, December, 1987). While MIRTE has been used 
in one recent study (Ackerman, 1987) to estimate item parameters, the author did not 
investigate questions considered in this study. 

Data Description 

Thrse different data sets were used (AirA2, A3) representing cases in which both 
underlying abilities (91 and 63) were normally distributed with mean 0, standard 
deviation 1 . The difference among the thre sets was the degiee of correlation between 
the abilities, namely 0.00, 0.25, and 0.50- 

The simulated test consisted of 104 items, 26 items requiring only the first ability, 
52 Items requiring predominantly the first ability, 26 items requiring equal amounts of 
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both abUitles, A listing of the item parameters Is provided In Table L Thirteen values 



of MDIF (rangln? from -3 to +3 at intervals of 0,5) and two values of MDISC (2,00, K70) 
were chosen in order to cover the range of difficulties and to simulate realistic 
discrimination conditions in which the items were designed to discriminate well on the 
first ability. To meet the requirement that the items discriminate well on the first 
ability, four values of the angle, aj^, (0*, 15^, 30^, 45*), were chosen. The 

discrimination indices, ai and a2 (one for each dimension)^ were then generated to fit 
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the corresponding d and MDISC. The correlations between the original item parameters 
were: p(d,ai) = 0.004; p(d,a2) = -0.004; p(aj,a2) = -0.738; and p(MDIF,MDISC) « -0.002. 



Because of the dependency of and a2, there is a larger correlation between these 

parameters. The same item parameters were used for each of the three data sets. 

Procedure 

The FORTRAN program M2PLGEN (Ackerman, 1985) was used to generate 2000 
ability vectors (e^, satisfying the distrlbuttons of e| and for Data Set A I. 

M2PLGEN uses a random seed and the IMSL (1979) subroutine GGNSM to generate 
random abilities. These ability vectors and the item parameters (a^, a2, d) were then 

used to generate response vectors (Os and U) for each of the 2000 simulees to each of 

the 104 items according to the M2PL modeL 

The 2000 x 104 matrix of response vectors was analyzed using MIRTE to provide 
estimates of 6^, $2, a^, 3^2* d, MDIF, MDISC, a^, and az^ These results were filed the 

random seed was inciemenied by two and the process was repeated. For Data Set AI 
there were 100 replications. Summary statistics were calculated on the lOO 
replications. 

This procedure was repeated tor the other data set conditions. The same initial 
item parameter estimates for a^ and a2 were used for every replication in order to 

provide better control in the design* Finally, summary results from the three data sets 
were compared* 



Results and Discussion 

The purpose of this research was to detecmine the effects of correlated abilities on 
parameter estimation given a two-dimensional, two-parameter logistic item response 
model. First it should be determined if suitable ability data were generated to model 
the condition*! specified. Then it needs to be determined whether MIRTE adequately 
estimated the parameters from the analysis of the response vectors generated. 

Generation of (ej^ r., Q^ ).: The ability data in all three data sets were generated to fit 
the specifications stated. The correlation between and 62 for data generated over 
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the 100 replications was recovered as -0.001 for Data Set Al, 0.251 for A2, and 0.500 
for A3. The means for and $2 range 0 i 0 004 and standard deviations 

-vere viihm U There vas very smaL "/ariancs ^iess :han 0.0C05) for 'hese 

means and standard deviations in all data sets. There were no replications in which 
the ability data were not satisfactorly generated. 

Recovery of Ability Parameters : In each of the three data .etsover the 100 
replications, 6^ and §2 had means of 0.00 and standard deviations of 1 .00. The 

standard deviation of the mean was less than O.OOl for all data sets. The recovery of 
these statistics is not particularly meaningful as a measure of accuracy in these cases 
because the MIRTE program rescales the theta estimates to mean 0, standard deviation 
1 after each iteration in order to prevent drifting of the esUmates. 

In the data analysis, the program doesn't always identify dimensions one and two 
correctly. In order to avoid confusing the dimensions during the 100 replications, a 
check was made during each data analysis on the first thirteen item discrimination 
parameter estimates. (These items were pure on 6^.) If the sum of the first thirteen 
ai estimates was less than the sxim of the first thirteen a2 estimates, the estimations 

for the dimensions were flipped. 

The mean average absolute deviation of 61 from the true 61 (AAD(ei)) ranged from 
0.44 to 0.46 (see Table 2). Increasing p(ei, 62) did not appear tc affect this. The mean 
average absolute deviation of 62 (AAD(^2^^ ranged from 0.54 to 0.41 and seemed to be 
more affected by the correlation between the abilities. As p(ei, 62) increased, the 
AAD(§2) decreased. This could be a result of the compensatory nature of the M2PL 
model. There was very little variance over replications in these AADs (0.001 for ^1; 
0.002 for §2) so that the thetas appear to have been recovered consistently across the 
three data sets. 

As 61 and 62 became more highly correlated, 62 appeared to be belter estimated 
(AAD(e2) decreased). This was supported by the mean correlation between 6^ and §2 
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As p(ei, 62) Increased, 62 became more highly correlated with 62 (Table 2). !n all three 
data sets» 63 was recovered fairly well according to r(e2, 62^- The mean standard 
error of the thetas (as calculated by MIRTE) was approximately 0.259, almost half the 
size of the AADs. The variance in these mean standard errors was very small 
although the standard errors were more spread out as the correlation between the 
dimensions increased. 
Table 2 

Mean Values of Statistics for Estimated Thetas (over 100 retjl lea t ions) 



Data 
Set 


p(ej,e2) 


AAD(ej) 


AAD(e2) 


r(ej,e2) 


r(ej,ei) 


r(e2,e2) 


r(ei,e2) 


r(e2,ej) 


Al 


0.00 


0.441 


0.544 


0.062 


0.842 


0.764 


0.505 


-0.295 


A2 


0.25 


0.446 


0.470 


0479 


0.842 


0.824 


0.603 


-0.050 


A3 


0.50 


0.459 


0.412 


0.282 


0.831 


0.865 


0.699 


-0.209 



The ability was also well recovered as rCSj, Sj) was greater than 0.83 for all 
three data sets. In Data Set A3, 62 appeared to be recovered better than 61 in spite of 
the lact that few items were measuring the e2-space. This was also supported by the 
decreasing AADCe^) as the correlation between the ability dimensions increased. As 
p(ei, 62) increased, 61 was less well recovered but 62 was better recovered. 

The correlation between the ability vectors was not well recovered. As pCe^, 62) 

increased, MIRTE tended to produce ability estimates which were less correlated than 
the generated abilities. The difference bet^we^;:; p(ei, 62) and rCe^, 62) increased as 
p(ei, 92^ increased. This result agrees with th:.t reported by Carlson (1987) 

Recovery of Iten^ pgramet ers: In the maximum likelihood estimation procedures 
used in MIRTE, ability estimates ara used to improve item parameter estimates and 
vice versa. Hence, the final estimates are affected by each other. As pCSj, 62) 

increased, what happened to the item parameter estimates? 
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statistics on the Item dlftlcultjr parameters are summarized in Table 3. In all three 
data sets, r(d, cl) ^ 0.997 indicating good estimation of the item difficulty. As pOj , 9^) 

increased, :he mean and scandara deviation of d increased slightly oui remained c!cse 
to the original parameter statistics. The AAD(d) increased slightly as the correlation 
between the ability dimensions increased indicating that ^ was being less well 
recovered. The mean and standard deviation of the multidimensional difficulty 
parameter, MDIF, were recovered well although here again MDIF was less well 
recovered as pv6i, 62) Increased. 

Table ^ 

Summarv of Mean Statistics for Item Dlfficultv (over 100 replications) 



Data 






3e(d) 












Set 


d 


s(d> 


AAD(d) 


MDIF 


s(MDIF) 


rCMDIF.MDIF) 


True 


0.009 


3.771 






-0.005 


2.058 






Al 


0.009 


3.929 


0.112 


0.224 


0.006 


2.079 


0.997 


0.995 


A2 


0.010 


3.936 


0.109 


0.228 


0.005 


2.028 


0.997 


0.994 


A3 


0.030 


3.936 


0.106 


0.232 


0.012 


1.999 


0.997 


0.991 



se - standard error from MIRTE program 



Discrimination parameter estimates have been reported to be affected more by 
multidimensional data and analysis. This result was also evident in this study. The 
mean of a^ was lower than the true mean and the standard deviation was higher than 
the true standard deviation for all three data sets (see Table 4). The mean of ^2 
much higher than the true mean of 0.678. In fact the mean of 33 was higher than the 
mean estimates of aj and approached the true mean of a^ as p(6p 63) increased Both 
means increased slightly as pCe^, 62) increased. The standard deviation of ^2 w^^s 

higher than the true standard deviation but there was not as large a difference here as 
withaj. Standard errors of estimation of aj and 33 were approximately 0.09 but the 
AADs were much larger, particularly for a2. As the correlation between the two 
ability dimensions increased, the AADCas) mcreased slightly indicating ^2 was being 
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less w^eil recovered. The AAD(at) was approximately 0.5 for all three data sets. The 
estimates of ai and a3 ranged over a much wider interval than the originals. 
Although In all three data sets the lower bound (0.010) was hit for some of both aj and 
a2 estimates, the upper bound of 4.500 w^as not hit until Data Set A3. 



Table 4 

Summary o f Mean St atistics for Item Discrimination (over 100 repUcations) 



Data 

Set ai s(ai) se(ai) AAD{ai) 3^2 ^^^2^ ^^^2> AAD(a2) MDISC s(MDISC) 



True 1.637 0.251 0.678 0.496 1.850 0.151 

AI 1.195 0.569 0.099 0.500 L379 0.512 0.096 0.707 1.957 0.288 

A2 1.201 0.528 0.095 0.4G6 L448 0.582 0.094 0.775 2.013 0.319 

A3 1.202 0.502 0.093 0.490 1.510 0.628 0.093 0.836 2.057 0.381 



The multidimensional discrimination parameter, MDISC, was recovered w^ith a 
higher mean and higher standard deviation in all three data sets. There appears to be 
a rotational Indeterminacy in the recovery of the discrimination parameters and a 
tendency to spread the discrimination parameter estimates over the entire spa^e even 
though they originally did not cover the entire space. 

This w^as supported by the statistics on the anjle estimates, and ^2-^ Originu'ly 
aj had a mean of 22.50*. This was recovered in all data sets at over 49''^ Similarly, 
a2, whose original mean was 67.50'', was recovered in all data sets at just over 40''. 

The original standard deviation of 16.85'' increased for the estimates to approximately 
20*. There seemed to be an attempt to cover the entire eie2-'Space in estin:^tion of 
parameters related to discrimination. Estimates of and a2 ranged from very close to 
0* to almost 90^. 

Correlation coefficients again were used to determine adequacy of parameter 
recovery (Table 5). In all cases, ai correlated more highly with a j than with a2. 
Similarly, a2 correlated more highly with a2 than it did with aj. As well, ^2 
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correlated higher with a2 than did with a2. The anomaly in the correlations was 
that a^ correlated less highly with a^ than a2 did with aj. As the discniAination 
paran *ers appear to be dispersed across the Qje^^space, this may account for the 
apparent better recovery of aj than of a^. That the standard deviation of a2 was * 
twice as large as that of may also account for the higher correlations of both a^ and 
a2 with a2> The greater variability in a2 would allow for higher correlations. 
Table 5 

Mean Correlations foiLltem Dlscrimlnatton Values (over 100 replications) 



Data Set 


r(ai,ai} 


r(a2,a2) 


r(ai,a2) 


r(ai^2' 


r(a2,ai) 


Al 


0.834 


0.893 


-0.765 


"0.572 


-0.865 


A2 


0.818 


0.899 


-0.769 


-0.587 


-0.830 


A3 


0.760 


0.895 


-0.735 


-0.587 


-0.747 



The correlation between aj and a2 was slightly stronger than the triie parameter 
correlation of -0.738 except in the Data Set A3 where it was slightly smaller. The 
multidimeni^ional discrimination parameter, MDISC, did not correl ate as highly with its 
estimate. This correlation was highest (0,600) when the ability dimensions were 
uncorrelated and decreased as the correlation between the abilities increased. 

The correlation between d and aj was 0,004. This was recovered as 0>019, 0.030, 
and 0,056 for Data Sets Al, A2, and A3 respectively. The correlation between d and a2 
was -0,004 which was recovered as 0,004, -0>011 and 0.004 for the three data sets 
respectively. This would suggest again that the second dimension is being better 
estimated. Thi*^. could be partially a result of the items of the simulated test not 
covering the entire latent space and the variability in a2 being so much greater than in 
a^ However, the AAD(a2) did not support the conclusion that a2 is better recovered 
than ai. The correlation between MDISC and MDIF was -0.002 and was recovered as 
-0.031, -0.027, and -0,073 for the three data sets respectively. There was no evi-'ent 
trend here> 

ERIC 



Conclusions 

The purpose of the study was to determine the effects of correlated dimensions in 
multidimensional estimation of data to fit the M2PLmoaeL , 

Results of this research support the use of MIRTE to analyse two-dimensional data 
fitting the M2PL model where the ability dimensions are normally distributed over a 
full range. Evea as the correlation between the ability dimensions increased from 0.00 
to 0.50, both item and ability parameters seemed to be recovered well enough to justify 
the use of a multidimensional analysis program rather than forcing unidimensionality 
by using one of the unidimensional analysis programs available. Correlation 
coefficients between parameters and corresponding estimates were high. 

A A A 

Average absolute deviations for S^^ 62 and aj were of a similar size. The AAD for 
32 vas much larger indicating the second dimension was not being as accurately 

recovered. As there were fewer items measuring the second dimension, recovery 
would be more difficult. The a^ discrimination parameters were slightly 
underestimated while the a2 discrimination parameters were overestimated. 

Item difficulty parameters have traditionally been recovered better than other 
parameters and results of this research continued to support this phenomenon. The 
AAD(S) was much smaller than for the other variables and the correlations between 
the difficulty parameters and estimates all had IH > 0.98. It should be remembered that 
that during the estimation procedures, to estimate item parameters there were 2000 
ability vectors, while the estimation of the ability vectors was done from estimates for 
only 104 i,ems. This would help account for better estimation of the single difficulty 
parameter. 

Both theta estimates were accurately estimated in terms of correlation. However, 

A A 

average absolute deviations of 0.44 to 0-46 for and 0.41 to 0.54 for 62 depending on 
the correlation between the ability dimensions were found. The correlation between 
the ability estimates for the two dimensions was less well recovered as the correlation 



between the ability dimensions increased. The correlation betTtreen the theta estimates 
was lover than the coi relation between the true thetas. This was in agreement with 
results reported by Carlson (:987), 

It would be useful to know how great this reduction of the correlation coefficient 
becomes as pCS], 63) inc*^ased beyond 0.50. As well, further research should consider 
the effects on parameter estimation when a], a], and a2 are generated to cover the 
entire space. It is possible that the anomalies associated with the recovery of a2 would 
be less severe gWen a different set of item characteristics. Conclusions based on the 
estimates cf parameters related to the second dimension in particular are tentative as 
it appears there was an attempt by MIRTE to estimate over the entire latent space. 
This seemed to have affected estimates on the second dimension moreso than those for 
the first ability dimension. 

Increasing the correlation between the true thetas did affect the e^^timation of 
parameters. The important parameter least well recovered in this study seemed to be 
a2^ This could be explained by the fact that there were fewer items measuring the 

second dimension. There is cause for concern over the size of the AADs for aoility and 
discrimination parameter estimates in all three data sets. 

The results of this research indicate a good future for the application of 
multidimensional models using the estimation procedures of MIRTE but additional 
studies are needed. 
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